Review: Lower Ed, by Tressie McMillan Cottom

Publisher:	The New Press
Copyright:	2017
Printing:	2018
ISBN:	1-62097-472-X
Format:	Kindle
Pages:	217

Lower Ed (subtitled The Troubling Rise of For-Profit Colleges in the New Economy) is the first book by sociologist Tressie McMillan Cottom. (I previously reviewed her second book, the excellent essay collection Thick.) It is a deep look at the sociology of for-profit higher education in the United States based on interviews with students and executives, analysis of Wall Street filings, tests of the admissions process, and her own personal experiences working for two of the schools. One of the questions that McMillan Cottom tries to answer is why students choose to enroll in these institutions, particularly the newer type of institution funded by federal student loans and notorious for being more expensive and less valuable than non-profit colleges and universities. I was hesitant to read this book because I find for-profit schools depressing. I grew up with the ubiquitous commercials, watched the backlash develop, and have a strongly negative impression of the industry, partly influenced by having worked in traditional non-profit higher education for two decades. The prevailing opinion in my social group is that they're a con job. I was half-expecting a reinforcement of that opinion by example, and I don't like reading infuriating stories about people being defrauded. I need not have worried. This is not that sort of book (nor, in retrospect, do I think McMillan Cottom would approach a topic from that angle). Sociology is broader than reporting. Lower Ed positions for-profit colleges within a larger social structure of education, credentialing, and changes in workplace expectations; takes a deep look at why they are attractive to their students; and humanizes and complicates the motives and incentives of everyone involved, including administrators and employees of for-profit colleges as well as the students. McMillan Cottom does of course talk about the profit motive and the deceptions surrounding that, but the context is less that of fraud that people are unable to see through and more a balancing of the drawbacks of a set of poor choices embedded in institutional failures. One of my metrics for a good non-fiction book is whether it introduces me to a new idea that changes how I analyze the world. Lower Ed does that twice. The first idea is the view of higher education through the lens of risk shifting. It used to be common for employers to hire people without prior job-specific training and do the training in-house, possibly through an apprenticeship structure. More notably, once one was employed by a particular company, the company routinely arranged or provided ongoing training. This went hand-in-hand with a workplace culture of long tenure, internal promotion, attempts to avoid layoffs, and some degree of mutual loyalty. Companies expected to invest significantly in an employee over their career and thus also had an incentive to retain that employee rather than train someone for a competitor. However, from a purely financial perspective, this is a risk and an inefficiency, similar to the risk of carrying a large inventory of parts and components. Companies have responded to investor-driven focus on profits and efficiency by reducing overhead and shifting risk. This leads to the lean supply chain, where no one pays for parts to sit around in warehouses and companies aren't caught with large stockpiles of now-useless components, but which is more sensitive to any disruption (such as from a global pandemic). And, for employment, it leads to a desire to hire pre-trained workers, retain only enough workers to do the current amount of work, and replace them with new workers who already have appropriate training rather than retrain them. The effect of the corporate decision to only hire pre-trained employees is to shift the risk and expense of training from the company to the prospective employee. The individual has to seek out training at their own expense in the hope (not guarantee) that at the conclusion of that training they will get or retain a job. People therefore turn to higher education to both provide that training and to help them decide what type of training will eventually be valuable. This has a long history with certain professional fields (doctors and lawyers, for example), but the requirements for completing training in those fields are relatively clear (a professional license to practice) and the compensation reflects the risk. What's new is the shift of training risk to the individual in more mundane jobs, without any corresponding increase in compensation. This, McMillan Cottom explains, is the background for the growth in demand for higher education in general and the the type of education offered by for-profit colleges in particular. Workers who in previous eras would be trained by their employers are now responsible for their own training. That training is no longer judged by the standards of a specific workplace, but is instead evaluated by a hiring process that expects constant job-shifting. This leads to increased demand by both workers and employers for credentials: some simple-to-check certificate of completion of training that says that this person has the skills to immediately start doing some job. It also leads to a demand for more flexible class hours, since the student is now often someone older with a job and a family to balance. Their ongoing training used to be considered a cost of business and happen during their work hours; now it is something they have to fit around the contours of their life because their employer has shifted that risk to them. The risk-shifting frame makes sense of the "investment" language so common in for-profit education. In this job economy, education as investment is not a weird metaphor for the classic benefits of a liberal arts education: broadened perspective, deeper grounding in philosophy and ethics, or heightened aesthetic appreciation. It's an investment in the literal financial sense; it is money that you spend now in order to get a financial benefit (a job) in the future. People have to invest in their own training because employers are no longer doing so, but still require the outcome of that investment. And, worse, it's primarily a station-keeping investment. Rather than an optional expenditure that could reap greater benefits later, it's a mandatory expenditure to prevent, at best, stagnation in a job paying poverty wages, and at worst the disaster of unemployment. This explains renewed demand for higher education, but why for-profit colleges? We know they cost more and have a worse reputation (and therefore their credentials have less value) than traditional non-profit colleges. Flexible hours and class scheduling explains some of this but not all of it. That leads to the second perspective-shifting idea I got from Lower Ed: for-profit colleges are very good at what they focus time and resources on, and they focus on enrolling students. It is hard to enroll in a university! More precisely, enrolling in a university requires bureaucracy navigation skills, and those skills are class-coded. The people who need them the most are the least likely to have them. Universities do not reach out to you, nor do they guide you through the process. You have to go to them and discover how to apply, something that is often made harder by the confusing state of many university web sites. The language and process is opaque unless other people in your family have experience with universities and can explain it. There might be someone you can reach on the phone to ask questions, but they're highly unlikely to proactively guide you through the remaining steps. It's your responsibility to understand deadlines, timing, and sequence of operations, and if you miss any of the steps (due to, for example, the overscheduled life of someone in need of better education for better job prospects), the penalty in time and sometimes money can be substantial. And admission is just the start; navigating financial aid, which most students will need, is an order of magnitude more daunting. Community colleges are somewhat easier (and certainly cheaper) than universities, but still have similar obstacles (and often even worse web sites). It's easy for people like me, who have long professional expertise with bureaucracies, family experience with higher education, and a support network of people to nag me about deadlines, to underestimate this. But the application experience at a for-profit college is entirely different in ways far more profound than I had realized. McMillan Cottom documents this in detail from her own experience working for two different for-profit colleges and from an experiment where she indicated interest in multiple for-profit colleges and then stopped responding before signing admission paperwork. A for-profit college is fully invested in helping a student both apply and get financial aid, devotes someone to helping them through that process, does not expect them to understand how to navigate bureaucracies or decipher forms on their own, does not punish unexpected delays or missed appointments, and goes to considerable lengths to try to keep anyone from falling out of the process before they are enrolled. They do not expect their students to already have the skills that one learns from working in white-collar jobs or from being surrounded by people who do. They provide the kind of support that an educational institution should provide to people who, by definition, don't understand something and need to learn. Reading about this was infuriating. Obviously, this effort to help people enroll is largely for predatory reasons. For-profit schools make their money off federal loans and they don't get that money unless they can get someone to enroll and fill out financial paperwork (and to some extent keep them enrolled), so admissions is their cash cow and they act accordingly. But that's not why I found it infuriating; that's just predictable capitalism. What I think is inexcusable is that nothing they do is that difficult. We could being doing the same thing for prospective community college students but have made the societal choice not to. We believe that education is valuable, we constantly advocate that people get more job training and higher education, and yet we demand prospective students navigate an unnecessarily baroque and confusing application process with very little help, and then stereotype and blame them for failing to do so. This admission support is not a question of resources. For-profit colleges are funded almost entirely by federally-guaranteed student loans. We are paying them to help people apply. It is, in McMillan Cottom's term, a negative social insurance program. Rather than buffering people against the negative effects of risk-shifting of employers by helping them into the least-expensive and most-effective training programs (non-profit community colleges and universities), we are spending tax dollars to enrich the shareholders of for-profit colleges while underfunding the alternatives. We are choosing to create a gap that routes government support to the institution that provides worse training at higher cost but is very good at helping people apply. It's as if the unemployment system required one to use payday lenders to get one's unemployment check. There is more in this book I want to talk about, but this review is already long enough. Suffice it to say that McMillan Cottom's analysis does not stop with market forces and the admission process, and the parts of her analysis that touch on my own personal experience as someone with a somewhat unusual college path ring very true. Speaking as a former community college student, the discussion of class credit transfer policies and the way that institutional prestige gatekeeping and the desire to push back against low-quality instruction becomes a trap that keeps students in the for-profit system deserves another review this length. So do the implications of risk-shifting and credentialism on the morality of "cheating" on schoolwork. As one would expect from the author of the essay "Thick" about bringing context to sociology, Lower Ed is personal and grounded. McMillan Cottom doesn't shy away from including her own experiences and being explicit about her sources and research. This is backed up by one of the best methodological notes sections I've seen in a book. One of the things I love about McMillan Cottom's writing is that it's solidly academic, not in the sense of being opaque or full of jargon (the text can be a bit dense, but I rarely found it hard to follow), but in the sense of being clear about the sources of knowledge and her methods of extrapolation and analysis. She brings her receipts in a refreshingly concrete way. I do have a few caveats. First, I had trouble following a structure and line of reasoning through the whole book. Each individual point is meticulously argued and supported, but they are not always organized into a clear progression or framework. That made Lower Ed feel at times like a collection of high-quality but somewhat unrelated observations about credentials, higher education, for-profit colleges, their student populations, their business models, and their relationships with non-profit schools. Second, there are some related topics that McMillan Cottom touches on but doesn't expand sufficiently for me to be certain I understood them. One of the big ones is credentialism. This is apparently a hot topic in sociology and is obviously important to this book, but it's referenced somewhat glancingly and was not satisfyingly defined (at least for me). There are a few similar places where I almost but didn't quite follow a line of reasoning because the book structure didn't lay enough foundation. Caveats aside, though, this was meaty, thought-provoking, and eye-opening, and I'm very glad that I read it. This is a topic that I care more about than most people, but if you have watched for-profit colleges with distaste but without deep understanding, I highly recommend Lower Ed. Rating: 8 out of 10

Keepalived is a Linux implementation of VRRP. The usual role of VRRP is to share a virtual IP across a set of routers. For each VRRP instance, a leader is elected and gets to serve the IP address, ensuring the high availability of the attached service. Keepalived can also be used for a generic leader election, thanks to its ability to use scripts for healthchecking and run commands on state change. A simple configuration looks like this: The state keyword in instructs Keepalived to not take the leader role when starting. Otherwise, incoming nodes create a temporary disruption by taking over the IP address until the election settles. The interface keyword in defines the interface for sending and receiving VRRP packets. It is also the default interface to configure the virtual IP address. The virtual_router_id directive in is common to all nodes sharing the virtual IP. The priority keyword in helps choosing which router will be elected as leader. If you need more information around Keepalived, be sure to check the documentation. VRRP design is tied to Ethernet networks and requires a multicast-enabled network for communication between nodes. In some environments, notably public clouds, multicast is unavailable. In this case, Keepalived can send VRRP packets using unicast: Another process, like a BGP daemon, should advertise the virtual IP address to the network . If needed, Keepalived can trigger whatever action is needed for this by using notify_* scripts. Until version 2.21 (not released yet), the interface directive is mandatory and Keepalived will transmit and receive VRRP packets on this interface only. If peers are reachable through several interfaces, like on a BGP on the host setup, you need a workaround. A simple one is to use a VXLAN interface: Learning of MAC addresses is disabled and one generic entry for each peer is added in the forwarding database: transmitted packets are broadcasted to all peers, notably VRRP packets. Have a look at VXLAN & Linux for additional details. Starting from Keepalived 2.21, unicast_peer can be used without the interface directive. I think using VXLAN is still a neat trick applicable to other situations where communication using broadcast or multicast is needed, while the underlying network provide no support for this.

The publisher MDPI is a spammer and lies. If you upload a paper draft to arXiv, MDPI will send spam to the authors to solicit submission. Within minutes of an upload I received the following email (sent by MDPI staff, not some overly eager new editor): First of all, the email begins with a lie. Because this paper clearly states that it is submitted elsewhere. Also, it fits other journals much better, and if they had read even just the abstract, they would have known. This is predatory behavior by MDPI. Clearly, it is just about getting as many submissions as possible. The journal charges 1000 CHF (next year, 1400 CHF) to publish the papers. Its about the money. Also, there have been reports that MDPI ignores the reviews, and always publishes even when reviewers recommended rejection The reviewer requests I have received from MDPI came with unreasonable deadlines, which will not allow for a thorough peer review. Hence I asked to not ever be emailed by them again. I must assume that many other qualified reviewers do the same. MDPI boasts in their 2019 annual report a median time to first decision of 19 days in my discipline the typical time window to ask for reviews is at least a month (for shorter conference papers, not full journal articles), because professors tend to have lots of other duties, hence they need more flexibility. Above paper has been submitted in March, and is now under review for 4 months already. This is an annoying long time window, and I would appreciate if this were less, but it shows how extremely short the MDPI time frame is. They also claim 269.1k submissions and 106.2k published papers, so the acceptance rate is around 40% on average, and assuming that there are some journals with higher standards there then some must have acceptance rates much higher than this. I d assume that many reputable journals have 40% desk-rejection rate for papers that are not even on-topic The average cost to authors is given as 1144 CHF (after discounts, 25% waived feeds etc.), so they, so we are talking about 120 million CHF of revenue from authors. Is that what you want academic publishing to be? I am not happy with some of the established publishers such as Elsevier that also overcharge universities heavily. I do think we need to change academic publishing, and arXiv is a big improvement here. But I do not respect publishers such as MDPI that lie and send spam.

Review: Golden Gates, by Conor Dougherty

Publisher:	Penguin
Copyright:	2020
ISBN:	0-525-56022-X
Format:	Kindle
Pages:	249

This review, for reasons that will hopefully become clear later, starts with a personal digression. I have been interested in political theory my entire life. That sounds like something admirable, or at least neutral. It's not. "Interested" means that I have opinions that are generally stronger than my depth of knowledge warrants. "Interested" means that I like thinking about and casting judgment on how politics should be done without doing the work of politics myself. And "political theory" is different than politics in important ways, not the least of which is that political actions have rarely been a direct danger to me or my family. I have the luxury of arguing about politics as a theory. In short, I'm at high risk of being one of those people who has an opinion about everything and shares it on Twitter. I'm still in the process (to be honest, near the beginning of the process) of making something useful out of that interest. I've had some success when I become enough a part of a community that I can do some of the political work, understand the arguments at a level deeper than theory, and have to deal with the consequences of my own opinions. But those communities have been on-line and relatively low stakes. For the big political problems, the ones that involve governments and taxes and laws, those that decide who gets medical treatment and income support and who doesn't, to ever improve, more people like me need to learn enough about the practical details that we can do the real work of fixing them, rather than only making our native (and generally privileged) communities better for ourselves. I haven't found my path helping with that work yet. But I do have a concrete, challenging, local political question that makes me coldly furious: housing policy. Hence this book. Golden Gates is about housing policy in the notoriously underbuilt and therefore incredibly expensive San Francisco Bay Area, where I live. I wanted to deepen that emotional reaction to the failures of housing policy with facts and analysis. Golden Gates does provide some of that. But this also turns out to be a book about the translation of political theory into practice, about the messiness and conflict that results, and about the difficult process of measuring success. It's also a book about how substantial agreement on the basics of necessary political change can still founder on the shoals of prioritization, tribalism, and people who are interested in political theory. In short, it's a book about the difficulty of changing the world instead of arguing about how to change it. This is not a direct analysis of housing policy, although Dougherty provides the basics as background. Rather, it's the story of the political fight over housing told primarily through two lenses: Sonja Trauss, founder of BARF (the Bay Area Renters' Federation); and a Redwood City apartment complex, the people who fought its rent increases, and the nun who eventually purchased it. Around that framework, Dougherty writes about the Howard Jarvis Taxpayers Association and the history of California's Proposition 13, a fight over a development in Lafayette, the logistics challenge of constructing sufficient housing even when approved, and the political career of Scott Wiener, the hated opponent of every city fighting for the continued ability to arbitrarily veto any new housing. One of the things Golden Gates helped clarify for me is that there are three core interest groups that have to be part of any discussion of Bay Area housing: homeowners who want to limit or eliminate local change, renters who are vulnerable to gentrification and redevelopment, and the people who want to live in that area and can't (which includes people who want to move there, but more sympathetically includes all the people who work there but can't afford to live locally, such as teachers, day care workers, food service workers, and, well, just about anyone who doesn't work in tech). (As with any political classification, statements about collectives may not apply to individuals; there are numerous people who appear to fall into one group but who vote in alignment with another.) Dougherty makes it clear that housing policy is intractable in part because the policies that most clearly help one of those three groups hurt the other two. As advertised by the subtitle, Dougherty's focus is on the fight for more housing. Those who already own homes whose values have been inflated by artificial scarcity, or who want to preserve such stratified living conditions as low-density, large-lot single-family dwellings within short mass-transit commute of one of the densest cities in the United States, don't get a lot of sympathy or focus here except as opponents. I understand this choice; I also don't have much sympathy. But I do wish that Dougherty had spent more time discussing the unsustainable promise that California has implicitly made to homeowners: housing may be impossibly expensive, but if you can manage to reach that pinnacle of financial success, the ongoing value of your home is guaranteed. He does mention this in passing, but I don't think he puts enough emphasis on the impact that a single huge, illiquid investment that is heavily encouraged by government policy has on people's attitude towards anything that jeopardizes that investment. The bulk of this book focuses on the two factions trying to make housing cheaper: Sonja Trauss and others who are pushing for construction of more housing, and tenant groups trying to manage the price of existing housing for those who have to rent. The tragedy of Bay Area housing is that even the faintest connection of housing to the economic principle of supply and demand implies that the long-term goals of those two groups align. Building more housing will decrease the cost of housing, at least if you build enough of it over a long enough period of time. But in the short term, particularly given the amount of Bay Area land pre-emptively excluded from housing by environmental protection and the actions of the existing homeowners, building more housing usually means tearing down cheap lower-density housing and replacing it with expensive higher-density housing. And that destroys people's lives. I'll admit my natural sympathy is with Trauss on pure economic grounds. There simply aren't enough places to live in the Bay Area, and the number of people in the area will not decrease. To the marginal extent that growth even slows, that's another tale of misery involving "super commutes" of over 90 minutes each way. But the most affecting part of this book was the detailed look at what redevelopment looks like for the people who thought they had housing, and how it disrupts and destroys existing communities. It's impossible to read those stories and not be moved. But it's equally impossible to not be moved by the stories of people who live in their cars during the week, going home only on weekends because they have to live too far away from their jobs to commute. This is exactly the kind of politics that I lose when I take a superficial interest in political theory. Even when I feel confident in a guiding principle, the hard part of real-world politics is bringing real people with you in the implementation and mitigating the damage that any choice of implementation will cause. There are a lot of details, and those details matter. Without the right balance between addressing a long-term deficit and providing short-term protection and relief, an attempt to alleviate unsustainable long-term misery creates more short-term misery for those least able to afford it. And while I personally may have less sympathy for the relatively well-off who have clawed their way into their own mortgage, being cavalier with their goals and their financial needs is both poor ethics and poor politics. Mobilizing political opponents who have resources and vote locally isn't a winning strategy. Dougherty is a reporter, not a housing or public policy expert, so Golden Gates poses problems and tells stories rather than describes solutions. This book didn't lead me to a brilliant plan for fixing the Bay Area housing crunch, or hand me a roadmap for how to get effectively involved in local politics. What it did do is tell stories about what political approaches have worked, how they've worked, what change they've created, and the limitations of that change. Solving political problems is work. That work requires understanding people and balancing concerns, which in turn requires a lot of empathy, a lot of communication, and sometimes finding a way to make unlikely allies. I'm not sure how broad the appeal of this book will be outside of those who live in the region. Some aspects of the fight for housing generalize, but the Bay Area (and I suspect every region) has properties specific to it or to the state of California. It has also reached an extreme of housing shortage that is rivaled in the United States only by New York City, which changes the nature of the solutions. But if you want to seriously engage with Bay Area housing policy, knowing the background explained here is nearly mandatory. There are some flaws I wish Dougherty would have talked more about traffic and transit policy, although I realize that could be another book but this is an important story told well. If this somewhat narrow topic is within your interests, highly recommended. Rating: 8 out of 10

Gibbsdavidl/CatterPlots

Did you ever wish you could make scatter plots with cat shaped points? Now you can! - Gibbsdavidl/CatterPlots

What is the best tool to use for drawing vector pictures? For me and probably for many others, the answer is pretty obvious: Illustrator, or, maybe, Inkscape.

A coloring book to help folks understand how SELinux works. - mairin/selinux-coloring-book

The EURion constellation (also known as Omron rings[1] or doughnuts[2]) is a pattern of symbols incorporated into a number of banknote designs worldwide since about 1996. It is added to help imaging software detect the presence of a banknote in a digital image. Such software can then block the user from reproducing banknotes to prevent counterfeiting using colour photocopiers. According to research from 2004, the EURion constellation is used for colour photocopiers but probably not used in computer software.[3] It has been reported that Adobe Photoshop will not allow editing of an image of a banknote, but in some versions this is believed to be due to a different, unknown digital watermark rather than the EURion constellation.[4][3]

This huge collection of non-scary optical illusions and fascinating visual phenomena emphasizes interactive exploration, beauty, and scientific explanation.

Generated photos are created from scratch by AI systems. All images can be used for any purpose without worrying about copyrights, distribution rights, infringement claims, or royalties.

Dokumentarfilm ber die Rangierer im Bahnhof Dresden-Friedrichstadt in der DDR aus dem Jahr 1984.

Il termine sardo femina accabadora, femina agabbad ra o, pi comunemente, agabbadora o accabadora (s'agabbad ra, lett. "colei che finisce", deriva dal sardo s'acabbu, "la fine" o dallo spagnolo acabar, "terminare") denota la figura storicamente incerta di una donna che si incaricava di portare la morte a persone di qualunque et , nel caso in cui queste fossero in condizioni di malattia tali da portare i familiari o la stessa vittima a richiederla. In realt non ci sono prove di tale pratica, che avrebbe riguardato alcune regioni sarde come Marghine, Planargia e Gallura[1]. La pratica non doveva essere retribuita dai parenti del malato poich il pagare per dare la morte era contrario ai dettami religiosi e della superstizione.

Alright the people have spoken and they want more cat genetics. So, I present to you all "Cat Coat Genetics 101: A Tweetorial", feat. pics of many real life cats (for science, of course...this baby is Caterpillar).

Debian.org added support for DKIM in 2020. To configure this on my side, I had to do the following, on top of my email configuration.

1. add this line to /etc/opendkim/signing.table:

   *@debian.org marcos-debian.anarcat.user
   

2. add this line to /etc/opendkim/key.table:

   marcos-debian.anarcat.user debian.org:marcos-debian.anarcat.user:/etc/opendkim/keys/marcos-debian.anarcat.user.private
   

   Yes, that's quite a mouthful! That magic selector is long in that way because it needs a special syntax (specifically the .anarcat.user suffix) for Debian to be happy. The -debian string is to tell me where the key is published. The marcos prefix is to remind me where the private is used.
3. generate the key with:

   opendkim-genkey --directory=/etc/opendkim/keys/ --selector=marcos-debian.anarcat.user --domain=debian.org --verbose
   

   This creates the DNS record in /etc/opendkim/keys/marcos-debian.anarcat.user.txt (alongside the private key in .key).
4. restart OpenDKIM:

   service opendkim restart
   

   The DNS record will look something like this:

   marcos-debian.anarcat.user._domainkey   IN  TXT ( "v=DKIM1; h=sha256; k=rsa; "
   "p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtKzBK2f8vg5yV307WAOatOhypQt3ANQ95iDaewkVehmx42lZ6b4PzA1k5DkIarxjkk+7m6oSpx5H3egrUSLMirUiMGsIb5XVGBPFmKZhDVmC7F5G1SV7SRqqKZYrXTufRRSne1eEtA31xpMP0B32f6v6lkoIZwS07yQ7DDbwA9MHfyb6MkgAvDwNJ45H4cOcdlCt0AnTSVndcl"
   "pci5/2o/oKD05J9hxFTtlEblrhDXWRQR7pmthN8qg4WaNI4WszbB3Or4eBCxhUdvAt2NF9c9eYLQGf0jfRsbOcjSfeus0e2fpsKW7JMvFzX8+O5pWfSpRpdPatOt80yy0eqpm1uQIDAQAB" )  ; ----- DKIM key marcos-debian.anarcat.user for debian.org
   

5. The "p=MIIB..." string needs to be joined together, without the quotes and the p=, and sent in a signed email to changes@db.debian.org:

   -----BEGIN PGP SIGNED MESSAGE-----
   dkimPubKey: marcos.anarcat.user MIIB[...]
   -----BEGIN PGP SIGNATURE-----
   [...]
   

6. Wait a few minutes for DNS to propagate. You can check if they have with:

   host -t TXT marcos-debian.anarcat.user._domainkey.debian.org nsp.dnsnode.net
   

   (nsp.dnsnode.net being one of the NS records of the debian.org zone.)

If all goes well, the tests should pass when sending from your server as anarcat@debian.org.

Testing Test messages can be sent to dkimvalidator, mail-tester.com or check-auth@verifier.port25.com. Those tools will run Spamassassin on the received emails and report the results. What you are looking for is:

• -0.1 DKIM_VALID: Message has at least one valid DKIM or DK signature
• -0.1 DKIM_VALID_AU: Message has a valid DKIM or DK signature from author's domain
• -0.1 DKIM_VALID_EF: Message has a valid DKIM or DK signature from envelope-from domain

If one of those is missing, then you are doing something wrong and your "spamminess" score will be worse. The latter is especially tricky as it validates the "Envelope From", which is the MAIL FROM: header as sent by the originating MTA, which you see as from=<> in the postfix lost. The following will happen anyways, as soon as you have a signature, that's normal:

• 0.1 DKIM_SIGNED: Message has a DKIM or DK signature, not necessarily valid

And this might happen if you have a ADSP record but do not correctly sign the message with a domain field that matches the record:

• 1.1 DKIM_ADSP_ALL No valid author signature, domain signs all mail

That's bad and will affect your spam core badly. I fixed that issue by using a wildcard key in the key table:

--- a/opendkim/key.table
+++ b/opendkim/key.table
@@ -1 +1 @@
-marcos anarc.at:marcos:/etc/opendkim/keys/marcos.private
+marcos %:marcos:/etc/opendkim/keys/marcos.private

References

• Ubuntu documentation
• Debian wiki
• linode tutorial, also recommends rotating keys every 6 months
• jak-linux: uses rspamd instead of opendkim, and PostSRSd
• Gio's tutorial

This is a copy of a subset of my more complete email configuration.

Gogo, the WiFi provider for airlines like Air Canada, is not available to Linux users even though it advertises "access using any Wi-Fi enabled laptop, tablet or smartphone". It is however possible to work-around this restriction by faking your browser user agent. I tried the User-Agent Switcher for Chrome extension on Chrome and Brave but it didn't work for some reason. What did work was using Firefox and adding the following prefs in about:config to spoof its user agent to Chrome for Windows:

general.useragent.override=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36
general.useragent.updates.enabled=false
privacy.resistFingerprinting=false

The last two prefs are necessary in order for the hidden general.useragent.override pref to not be ignored.

Opt out of mandatory arbitration As an aside, the Gogo terms of service automatically enroll you into mandatory arbitration unless you opt out by sending an email to customercare@gogoair.com within 30 days of using their service. You may want to create an email template for this so that you can fire off a quick email to them as soon as you connect. I will probably write a script for it next time I use this service.

Review: Thick, by Tressie McMillan Cottom

Publisher:	The New Press
Copyright:	2019
ISBN:	1-62097-437-1
Format:	Kindle
Pages:	247

Tressie McMillan Cottom is an associate professor of sociology at Virginia Commonwealth University. I first became aware of her via retweets and recommendations from other people I follow on Twitter, and she is indeed one of the best writers on that site. Thick: And Other Essays is an essay collection focused primarily on how American culture treats black women. I will be honest here, in part because I think much of the regular audience for my book reviews is similar to me (white, well-off from working in tech, and leftist but privileged) and therefore may identify with my experience. This is the sort of book that I always want to read and then struggle to start because I find it intimidating. It received a huge amount of praise on release, including being named as a finalist for the National Book Award, and that praise focused on its incisiveness, its truth-telling, and its depth and complexity. Complex and incisive books about racism are often hard for me to read; they're painful, depressing, and infuriating, and I have to fight my tendency to come away from them feeling more cynical and despairing. (Despite loving his essays, I'm still procrastinating reading Ta-Nehisi Coates's books.) I want to learn and understand but am not good at doing anything with the information, so this reading can feel like homework. If that's also your reaction, read this book. I regret having waited as long as I did. Thick is still, at times, painful, depressing, and infuriating. It's also brilliantly written in a way that makes the knowledge being conveyed easier to absorb. Rather than a relentless onslaught of bearing witness (for which, I should stress, there is an important place), it is a scalpel. Each essay lays open the heart of a subject in a few deft strokes, points out important features that the reader has previously missed, and then steps aside, leaving you alone with your thoughts to come to terms with what you've just learned. I needed this book to be an essay collection, with each thought just long enough to have an impact and not so long that I became numb. It's the type of collection that demands a pause at the end of each essay, a moment of mental readjustment, and perhaps a paging back through the essay again to remember the sharpest points. The essays often start with seeds of the personal, drawing directly on McMillan Cottom's own life to wrap context around their point. In the first essay, "Thick," she uses advice given her younger self against writing too many first-person essays to talk about the writing form, its critics, and how the backlash against it has become part of systematic discrimination because black women are not allowed to write any other sort of authoritative essay. She then draws a distinction between her own writing and personal essays, not because she thinks less of that genre but because that genre does not work for her as a writer. The essays in Thick do this repeatedly. They appear to head in one direction, then deepen and shift with the added context of precise sociological analysis, defying predictability and reaching a more interesting conclusion than the reader had expected. And, despite those shifts, McMillan Cottom never lost me in a turn. This is a book that is not only comfortable with complexity and nuance, but helps the reader become comfortable with that complexity as well. The second essay, "In the Name of Beauty," is perhaps my favorite of the book. Its spark was backlash against an essay McMillan Cottom wrote about Miley Cyrus, but the topic of the essay wasn't what sparked the backlash.

What many black women were angry about was how I located myself in what I'd written. I said, blithely as a matter of observable fact, that I am unattractive. Because I am unattractive, the argument went, I have a particular kind of experience of beauty, race, racism, and interacting with what we might call the white gaze. I thought nothing of it at the time I was writing it, which is unusual. I can usually pinpoint what I have said, written, or done that will piss people off and which people will be pissed off. I missed this one entirely.

What follows is one of the best essays on the social construction of beauty I've ever read. It barely pauses at the typical discussion of unrealistic beauty standards as a feminist issue, instead diving directly into beauty as whiteness, distinguishing between beauty standards that change with generations and the more lasting rules that instead police the bounds between white and not white. McMillan Cottom then goes on to explain how beauty is a form of capital, a poor and problematic one but nonetheless one of the few forms of capital women have access to, and therefore why black women have fought to be included in beauty despite all of the problems with judging people by beauty standards. And the essay deepens from there into a trenchant critique of both capitalism and white feminism that is both precise and illuminating.

When I say that I am unattractive or ugly, I am not internalizing the dominant culture's assessment of me. I am naming what has been done to me. And signaling who did it. I am glad that doing so unsettles folks, including the many white women who wrote to me with impassioned cases for how beautiful I am. They offered me neoliberal self-help nonsense that borders on the religious. They need me to believe beauty is both achievable and individual, because the alternative makes them vulnerable.

I could go on. Every essay in this book deserves similar attention. I want to quote from all of them. These essays are about racism, feminism, capitalism, and economics, all at the same time. They're about power, and how it functions in society, and what it does to people. There is an essay about Obama that contains the most concise explanation for his appeal to white voters that I've read. There is a fascinating essay about the difference between ethnic black and black-black in U.S. culture. There is so much more.

We do not share much in the U.S. culture of individualism except our delusions about meritocracy. God help my people, but I can talk to hundreds of black folks who have been systematically separated from their money, citizenship, and personhood and hear at least eighty stories about how no one is to blame but themselves. That is not about black people being black but about people being American. That is what we do. If my work is about anything it is about making plain precisely how prestige, money, and power structure our so-called democratic institutions so that most of us will always fail.

I, like many other people in my profession, was always more comfortable with the technical and scientific classes in college. I liked math and equations and rules, dreaded essay courses, and struggled to engage with the mandatory humanities courses. Something that I'm still learning, two decades later, is the extent to which this was because the humanities are harder work than the sciences and I wasn't yet up to the challenge of learning them properly. The problems are messier and more fluid. The context required is broader. It's harder to be clear and precise. And disciplines like sociology deal with our everyday lived experience, which means that we all think we're entitled to an opinion. Books like this, which can offer me a hand up and a grounding in the intellectual rigor while simultaneously being engaging and easy to read, are a treasure. They help me fill in the gaps in my education and help me recognize and appreciate the depth of thought in disciplines that don't come as naturally to me. This book was homework, but the good kind, the kind that exposes gaps in my understanding, introduces topics I hadn't considered, and makes the time fly until I come up for air, awed and thinking hard. Highly recommended. Rating: 9 out of 10

A new (upstream) simdjson release was announced by Daniel Lemire earlier this week, and my Twitter mentions have been running red-hot ever since as he was kind enough to tag me. Do look at that blog post, there is some impressive work in there. We wrapped up the (still very simple) rcppsimdjson around it last night and shipped it this morning. RcppSimdJson wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire. Via some very clever algorithmic engineering to obtain largely branch-free code, coupled with modern C++ and newer compiler instructions, it results in parsing gigabytes of JSON parsed per second which is quite mindboggling. For illustration, I highly recommend the video of the recent talk by Daniel Lemire at QCon (which was also voted best talk). The best-case performance is faster than CPU speed as use of parallel SIMD instructions and careful branch avoidance can lead to less than one cpu cycle use per byte parsed. This release brings upstream 0.3 (and 0.3.1) plus a minor tweak (also shipped back upstream). Our full NEWS entry follows.

Changes in version 0.0.4 (2020-04-03)

• Upgraded to new upstream releases 0.3 and 0.3.1 (Dirk in #9 closing #8)
• Updated example validateJSON to API changes.

But because Daniel is such a fantastic upstream developer to collaborate with, he even filed a full feature-request maybe you can consider upgrading as issue #8 at our repo containing the fully detailed list of changes. As it is so impressive I will simple quote the upper half of just the major changes:

Highlights

• Multi-Document Parsing: Read a bundle of JSON documents (ndjson) 2-4x faster than doing it individually. API docs / Design Details
• Simplified API: The API has been completely revamped for ease of use, including a new JSON navigation API and fluent support for error code and exception styles of error handling with a single API. Docs
• Exact Float Parsing: Now simdjson parses floats flawlessly without any performance loss (https://github.com/simdjson/simdjson/pull/558). Blog Post
• Even Faster: The fastest parser got faster! With a shiny new UTF-8 validator and meticulously refactored SIMD core, simdjson 0.3 is 15% faster than before, running at 2.5 GB/s (where 0.2 ran at 2.2 GB/s).

For questions, suggestions, or issues please use the issue tracker at the GitHub repo. Courtesy of CRANberries, there is also a diffstat report for this release. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Here's what happened in the Reproducible Builds effort between Sunday October 22 and Saturday October 28 2017: Past Events

• On Tuesday 24th October, Chris Lamb presented at All Things Open 2017 in Raleigh, NC, USA.
• On Wednesday 25th October, Holger Levsen presented at the Open Source Summit Europe in Prague, Czech Republic.
• On Saturday 28th October, Chris Lamb presented at freenode.live in Bristol, UK.

Upcoming/current events

• From October 31st November 2nd we will be holding the 3rd Reproducible Builds summit in Berlin, Germany.
• On November 8th Jonathan Bustillos Osornio (jathan) will present at CubaConf Havana.

Documentation updates Bernhard Wiedemann started The Unreproducible Package which "is meant as a practical way to demonstrate the various ways that software can break reproducible builds using just low level primitives without requiring external existing programs that implement these primitives themselves. It is structured so that one subdirectory demonstrates one class of issues in some variants observed in the wild." Reproducible work in other projects Hush, a fork of ZCash, opened an issue into reproducible builds. A new tag was added to lintian (lint checker for Debian packages) to ensure that changelog entry timestamps are strictly increasing. This avoids certain real-world issues with identical timestamps, documented in Debian #843773. Packages reviewed and fixed, and bugs filed Patches sent upstream:

• Bernhard M. Wiedemann:
  • gtranslator, embedded build timestamps
  • libgda, embedded build timestamps
  • mariadb, embedded build timestamps
  • nim, embedded build timestamps

Debian bug reports:

• Chris Lamb:
  • #879569 filed against argagg.
  • #879913 filed against sdlgfx.
  • #879914 filed against fmtlib.

Reviews of unreproducible packages 14 package reviews have been added, 35 have been updated and 28 have been removed in this week, adding to our knowledge about identified issues. 1 issue type has been updated:

• Add nondeterministic_ordering_in_documentation_generated_by_doxygen.

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (4)

strip-nondeterminism development Version 0.040-1 was uploaded to unstable by Mattia Rizzolo. It included contributions already covered by posts of the previous weeks, as well as new ones from:

• Mattia Rizzolo:
  • png.pm: Don't open the original file in write mode

reprotest development Development continued in git:

• Ximin Luo:
  • New features:
    • Support a domain_host variation.
    • Support a --print-sudoers feature.
  • Documentation:
    • Note some caveats about the existing git versions as a self-reminder not to release it yet.
    • Updates about our assumptions and rearrange sudo into its own section.
  • Bug fixes:
    • main: When dropping privs, make sure the user can still move in theroot.
    • tests: fix, need to preserve env for su
    • build: Don't fail when the build produces a broken symlink
    • main, presets: Properly drop privs when running the build. (Closes: #877813)
  • Code quality:
    • Improve logging to try to get to the bottom of the jenkins failures
    • Tweak tests to avoid some build dependencies
    • build: Name temporary directories after reprotest not autopkgtest

buildinfo.debian.net development Development continued in git:

• Chris Lamb:
  • New features:
    • Add API endpoint to fetch specific .buildinfo files for a certain package/version/architecture, and optimise it. (Closes: #25)
  • Bug fixes:
    • Always show SHA256, regardless of viewport size. (Closes: #27)
    • Actually filter by source package

reproducible-website development

• Holger Levsen:
  • RWS3 Berlin 2017:
    • Add CoyIM, Arch Linux, LEDE, LEAP, subuser.org, Bazel, coreboot.
    • Make some sponsors visible.
    • Add short paragraph explaining that registration is mandatory.

Misc. This week's edition was written by Ximin Luo, Chris Lamb, Bernhard M. Wiedemann and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Welcome to gambaru.de. Here is my monthly report that covers what I have been doing for Debian. If you re interested in Java, Games and LTS topics, this might be interesting for you. Debian Games

• I sponsored a new release of hexalate for Unit193 and icebreaker for Andreas Gnau. The latter is a reintroduction.
• New upstream releases this month: freeorion and hyperrogue.
• I backported freeciv and freeorion to Stretch.

Debian Java

• New upstream releases and one update: sweethome3d, sweethome3d-furniture, sweethome3d-furniture-editor, sweethome3d-textures-editor (update), libsambox-java, libsejda-java, pdfsam, easymock, jboss-modules, jboss-xnio and undertow.
• I fixed one RC bug in libsejda-io-java (#874494) and investigated another one (#869266) in commons-httpclient which could be closed.
• The new build-dependencies of jboss-xnio, wildfly-client-config and wildfly-common, were accepted into the archive this month.
• I spent some quality time on fixing #874579 in libhibernate-validator-java. This was the last blocking bug for pdfsam which I could finally upload to unstable. It s a really great JavaFX application. Check it out!
• I sponsored another update of libimglib2-java for Ghislain Vaillant and simplyhtml, freeplane and knopflerfish-osgi for Felix Natter.
• I also fixed RC bug #871348 in robocode, a Java programming game and #871347 in tycho.

Debian LTS This was my nineteenth month as a paid contributor and I have been paid to work 15,75 hours on Debian LTS, a project started by Rapha l Hertzog. In that time I did the following:

• From 18. September to 24. September I was in charge of our LTS frontdesk. I triaged bugs in poppler, binutils, kannel, wordpress, libsndfile, libexif, nautilus, libstruts1.2-java, nvidia-graphics-drivers, p3scan, otrs2 and glassfish.
• DLA-1108-1. Issued a security update for tomcat7 fixing 1 CVE.
• DLA-1116-1. Issued a security update for poppler fixing 3 CVE.
• DLA-1119-1. Issued a security update for otrs2 fixing 4 CVE.
• DLA-1122-1. Issued a security update for asterisk fixing 1 CVE. I also investigated CVE-2017-14099 and CVE-2017-14603. I decided against a backport because the fix was too intrusive and the vulnerable option is disabled by default in Wheezy s version which makes it a minor issue for most users.
• I submitted a patch for Debian s reportbug tool. (#878088) During our LTS BoF at DebConf 17 we came to the conclusion that we should implement a feature in reportbug that checks whether the bug reporter wants to report a regression for a recent security update. Usually the LTS and security teams receive word from the maintainer or users who report issues directly to our mailing lists or IRC channels. However in some cases we were not informed about possible regressions and the new feature in reportbug shall ensure that we can respond faster to such reports.
• I started to investigate the open security issues in wordpress and will complete the work in October.

Misc

• I packaged a new version of xarchiver. Thanks to the work of Ingo Br ckl xarchiver can handle almost all archive formats in Debian now.

QA upload

• I did a QA upload of xball, an ancient game from the 90ies that simulates bouncing balls. It should be ready for another decade at least.

Thanks for reading and see you next time.

Changes

• myrepos: mention related software
• icns: OpenJPEG 2 fixes (1 2 3 4), fix minor buffer overrun
• check-all-the-things: fix cwd usage, dependencies updates (1, 2), increased verbosity, TODO items (for errcheck, coccinelle, DR.CHECKER, RATS, qmllint, xmlpatternsvalidator, gfapy-validate, phpstan, psalm, ansible-lint)
• spelling dictionaries:
  • codespell: delte, defualt, Debiab, aiffer, syntesis, reprociblbe, thether, suport, mimimum, isssue, reseting
  • lintian: delte, Debiab, aiffer, syntesis, reprociblbe
• Debian Planets: add BCCD, Pardus
• Debian derivatives census: improve db handling
• Debian PTS: fix backports issue
• Debian member portfolio: https (1, 2, 3), drop alioth files, fix duck & lists search
• Debian security tracker: BlueBourne NFUs, u-boot issues, kanboard CVE list fixes, fix version typo
• Debian puppet: SSL exception (1, 2), comment fixes
• Debian website: add missing section title
• Debian wiki pages: AutomaticPackagingTools, CrossCompiling, DebianDevelopment, DebianLive/buster, DebianYeeloong/ko, Derivatives/Census (QA, BCCD, EmmabuntusDE (1, 2), ev3dev, Netrunner (1, 2), Pardus), DeveloperNews, Diagrams, EmDebian/CrossDebootstrap, InstallingDebianOn/Asus/Transformer T300 Chi/Stretch, IRC, JamesAmontgomery, JensKorte/reportbug-by-mail, LocalGroupsTemplate, Mobile, MultimediaFetchingTools, OfflineMasterKey (1, 2), Outreachy/Round15 (Projects/SocialEventAndConferenceCalendars), PaulWise/InterestingSoftware, PortTemplate, qa.debian.org, research/publications, RISC-V, Sprints/2017/DebianCloudOct2017, SystemBuildTools, Teams (Dpkg/FAQ, Publicity/micronews, ReleaseTeam/ReleaseCheckList), template
• File Formats wiki pages: ZED (1, 2)

Issues

• Crashes in firefox-esr, lugaru, minetest-mod-animals
• Conffile removal in fonts-beng-extra
• Upstream renamed openggsn
• Strictness in ftp.d.o
• Features in unattended-upgrades
• Missing hashes in caffeine-developers PPA
• Missing conffile removal in di-netboot-assistant
• Incorrect upgrades in unattended-upgrades
• Incorrect URL in reproducible-check
• Verbosity in github-backup
• Minor issues in unattended-upgrades
• Usability issue in Firefox
• Upstreaming needed for Hone Linux-Sensor
• New upstream version of samba

Review

• Spam: reported 1 LWN comment, 5 Debian bug reports and 338 Debian mailing list posts
• Debian packages: sponsored minetest-mod-unifieddyes 20170925-1
• Debian wiki: RecentChanges for the month
• Debian screenshots:
  • approved banshee, blender, bsdgames, chromium, eficas, elpa-visual-fill-column, elpa-writegood-mode, glirc, gnome-builder, gnome-shell-extension-taskbar, gwenview, knocker, krita, ksysguard, masscan, mediainfo-gui, mousepad, mysql-workbench, nitrogen, ola, pavucontrol, qupzilla, tiger, transmission-gtk, virtualbox, vlc, wavemon, wfuzz
  • deleted mysql-workbench (ugly, Windows), udisks2 (of other software)
  • rejected elpa-* (MacOS and taken from upstream), 0ad (RPi logo), synaptic (weird overlaid stuff), dehydrated (hilarious but not a screenshot), di-netboot-assistant (of other software), wfuzz (help output)
  • approved deletion of udisks2 (of other software), scottfree (manual page), agenda.app (of other software)
  • rejected deletion of reportbug/abe/nlkt (bogus reason)

Administration

• icns: merged patches
• Debian: help guest user with access, investigate/escalate broken network, restart broken stunnels, investigate static.d.o storage, investigate weird RAID mails, ask hoster to investigate power issue,
• Debian mentors: lintian/security updates & reboot
• Debian wiki: merged & deployed patch, redirect DDTSS translator, redirect user support requests, whitelist email addresses, update email for accounts with bouncing email,
• Debian derivatives census: merged/deployed patches
• Debian PTS: debugged cron mails, deployed changes, reran scripts, fixed configuration file
• Openmoko: debug reboot issue, debug load issues

Communication

• Inform derivatives that are using Alioth about the pending shutdown
• Inquire about Netrunner, Pardus in the Debian derivatives census.
• Invite OB2D Linux to the Debian derivatives census
• Welcome Rohan Garg (of Netrunner), Aaron Weeden (of BCCD), Muhammet Kara (of Pardus) to the Debian derivatives census

Sponsors The samba bug was sponsored by my employer. All other work was done on a volunteer basis.

Welcome to gambaru.de. Here is my monthly report that covers what I have been doing for Debian. If you re interested in Java, Games and LTS topics, this might be interesting for you. DebConf 17 in Montreal I traveled to DebConf 17 in Montreal/Canada. I arrived on 04. August and met a lot of different people which I only knew by name so far. I think this is definitely one of the best aspects of real life meetings, putting names to faces and getting to know someone better. I totally enjoyed my stay and I would like to thank all the people who were involved in organizing this event. You rock! I also gave a talk about the The past, present and future of Debian Games , listened to numerous other talks and got a nice sunburn which luckily turned into a more brownish color when I returned home on 12. August. The only negative experience I made was with my airline which was supposed to fly me home to Frankfurt again. They decided to cancel the flight one hour before check-in for unknown reasons and just gave me a telephone number to sort things out. No support whatsoever. Fortunately (probably not for him) another DebConf attendee suffered the same fate and together we could find another flight with Royal Air Maroc the same day. And so we made a short trip to Casablanca/Morocco and eventually arrived at our final destination in Frankfurt a few hours later. So which airline should you avoid at all costs (they still haven t responded to my refund claims) ? It s WoW-Air from Iceland. (just wow) Debian Games

• There were a lot of GCC-7 bugs to fix this month which claimed most of my games related time.
• Bug fixes: torcs (RC #853685), berusky2 (RC #853325), lordsawar (RC #853529), simutrans(#869029), gngb (RC #853425), libclaw (RC #853488), funguloids (RC #853408), plee-the-bear (RC #853618), alien-arena (RC #871218), fretsonfire (RC #872934), minetest, (RC #873324) widelands, (RC #871114), pingus (RC #853614)
• I released version 2.1 of the Debian Games Blend.
• I completely overhauled the gngb package, a color gameboy emulator.
• I packaged new upstream versions of freeciv, peg-e and blockattack.
• I backported the memory leak fix for unknown-horizons and fife to Stretch (#871037).
• I investigated some graphical glitches in Neverball which appear to be related to OpenGL and the graphic stack in general but I couldn t find an immediate solution. (#871223)

Debian Java

• I sponsored libimglib2-java for Ghislain Vaillant.
• New upstream releases: apktool, jboss-modules, jboss-logging-tools, jboss-logmanager.
• For jboss-xnio I packaged two new build-dependencies which are wildfly-common and wildfly-client-config and they are currently waiting in the NEW queue.
• The last build-dependency for PDFsam was accepted this month and I was able to upload the new version to experimental. Unfortunately the program is currently not really usable due to a bug in libhibernate-validator-java (#874579)

Debian LTS This was my eighteenth month as a paid contributor and I have been paid to work 20,25 hours on Debian LTS, a project started by Rapha l Hertzog. In that time I did the following:

• From 31. July until 06. August I was in charge of our LTS frontdesk. I triaged bugs in tinyproxy, mantis, sox, timidity, ioquake3, varnish, libao, clamav, binutils, smplayer, libid3tag, mpg123 and shadow.
• DLA-1064-1. Issued a security update for freeradius fixing 6 CVE.
• DLA-1068-1. Issued a security update for git fixing 1 CVE.
• DLA-1077-1. Issued a security update for faad2 fixing 11 CVE.
• DLA-1083-1. Issued a security update for openexr fixing 3 CVE.
• DLA-1095-1. Issued a security update for freerdp fixing 5 CVE.

Non-maintainer upload

• I uploaded a security fix for openexr (#864078) to fix CVE-2017-9110, CVE-2017-9112 and CVE-2017-9116.

Thanks for reading and see you next time.

TL,DR: I reverse-engineered the .zed encrypted archive format.
Following a clean-room design, I'm providing a description that can be implemented by a third-party.
Interested? (reference version at: https://www.beuc.net/zed/) .zed archive file format Introduction Archives with the .zed extension are conceptually similar to an encrypted .zip file. In addition to a specific format, .zed files support multiple users: files are encrypted using the archive master key, which itself is encrypted for each user and/or authentication method (password, RSA key through certificate or PKCS#11 token). Metadata such as filenames is partially encrypted. .zed archives are used as stand-alone or attached to e-mails with the help of a MS Outlook plugin. A variant, which is not covered here, can encrypt/decrypt MS Windows folders on the fly like ecryptfs. In the spirit of academic and independent research this document provides a description of the file format and encryption algorithms for this encrypted file archive. See the conventions section for conventions and acronyms used in this document. Structure overview The .zed file format is composed of several layers.

• The main container is using the (MS-CFB), which is notably used by MS Office 97-2003 .doc files. It contains several streams:
  • Metadata stream: in OLE Property Set format (MS-OLEPS), contains 2 blobs in a specific Type-Length-Value (TLV) format:
    • _ctlfile: global archive properties and access list
      It is obfuscated by means of static-key AES encryption.
      The properties include archive initial filename and a global IV.
      A global encryption key is itself encrypted in each user entry.
    • _catalog: file list
      Contains each file metadata indexed with a 15-bytes identifier.
      Directories are supported.
      Full filename is encrypted using AES.
      File extension is (redundantly) stored in clear, and so are file metadata such as modification time.
  • Each file in the archive compressed with zlib and encrypted with the standard AES algorithm, in a separate stream.
    Several encryption schemes and key sizes are supported.
    The file stream is split in chunks of 512 bytes, individually encrypted.
  • Optional streams, contain additional metadata as well as pictures to display in the application background ("watermarks"). They are not discussed here.

Or as a diagram:

+----------------------------------------------------------------------------------------------------+
  .zed archive (MS-CBF)                                                                               
                                                                                                      
   stream #1                         stream #2                       stream #3...                     
  +------------------------------+  +---------------------------+  +---------------------------+      
    metadata (MS-OLEPS)               encryption (AES)               encryption (AES)                 
                                      512-bytes chunks               512-bytes chunks                 
    +--------------------------+                                                                      
      obfuscation (static key)        +-----------------------+      +-----------------------+        
      +----------------------+       -  compression (zlib)     -    -  compression (zlib)     -       
       _ctlfile (TLV)                                                                            ...  
      +----------------------+          +---------------+              +---------------+               
    +--------------------------+          file contents                  file contents                
                                                                                                      
    +--------------------------+     -  +---------------+      -    -  +---------------+      -       
      _catalog (TLV)                                                                                  
    +--------------------------+      +-----------------------+      +-----------------------+        
  +------------------------------+  +---------------------------+  +---------------------------+      
+----------------------------------------------------------------------------------------------------+

Encryption schemes Several AES key sizes are supported, such as 128 and 256 bits. The Cipher Block Chaining (CBC) block cipher mode of operation is used to decrypt multiple AES 16-byte blocks, which means an initialisation vector (IV) is stored in clear along with the ciphertext. All filenames and file contents are encrypted using the same encryption mode, key and IV (e.g. if you remove and re-add a file in the archive, the resulting stream will be identical). No cleartext padding is used during encryption; instead, several end-of-stream handlers are available, so the ciphertext has exactly the size of the cleartext (e.g. the size of the compressed file). The following variants were identified in the 'encryption_mode' field. STREAM This is the end-of-stream handler for:

• obfuscated metadata encrypted with static AES key
• filenames and files in archives with 'encryption_mode' set to "AES-CBC-STREAM"
• any AES ciphertext of size < 16 bytes, regardless of encryption mode

This end-of-stream handler is apparently specific to the .zed format, and applied when the cleartext's does not end on a 16-byte boundary ; in this case special processing is performed on the last partial 16-byte block. The encryption and decryption phases are identical: let's assume the last partial block of cleartext (for encryption) or ciphertext (for decryption) was appended after all the complete 16-byte blocks of ciphertext:

• the second-to-last block of the ciphertext is encrypted in AES-ECB mode (i.e. block cipher encryption only, without XORing with the IV)
• then XOR-ed with the last partial block (hence truncated to the length of the partial block)

In either case, if the full ciphertext is less then one AES block (< 16 bytes), then the IV is used instead of the second-to-last block. CTS CTS or CipherText Stealing is the end-of-stream handler for:

• filenames and files in archives with 'encryption_mode' set to "AES-CBC-CTS".
  • exception: if the size of the ciphertext is < 16 bytes, then "STREAM" is used instead.

It matches the CBC-CS3 variant as described in Recommendation for Block Cipher Modes of Operation: Three Variants of Ciphertext Stealing for CBC Mode. Empty cleartext Since empty filenames or metadata are invalid, and since all files are compressed (resulting in a minimum 8-byte zlib cleartext), no empty cleartext was encrypted in the archive. metadata stream It is named 05356861616161716149656b7a6565636e576a33317a7868304e63 (hexadecimal), i.e. the character with code 5 followed by '5haaaaqaIekzeecnWj31zxh0Nc' (ASCII). The format used is OLE Property Set (MS-OLEPS). It introduces 2 property names "_ctlfile" (index 3) and "_catalog" (index 4), and 2 instances of said properties each containing an application-specific VT_BLOB (type 0x0041). _ctlfile: obfuscated global properties and access list This subpart is stored under index 3 ("_ctlfile") of the MS-OLEPS metadata. It consists of:

• static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by 0100 (hexadecimal) (18 bytes total)
• 16-byte IV
• ciphertext
• 1 uint32be representing the length of all the above
• static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by "ZoneCentral (R)" (ASCII) and a NUL byte (32 bytes total)

The ciphertext is encrypted with AES-CBC "STREAM" mode using 128-bit static key 37F13CF81C780AF26B6A52654F794AEF (hexadecimal) and the prepended IV so as to obfuscate the access list. The ciphertext is continuous and not split in chunks (unlike files), even when it is larger than 512 bytes. The decrypted text contain properties in a TLV format as described in _ctlfile TLV:

• global archive properties as a 'fileprops' structure,
• extra archive properties as a 'archive_extraprops' structure
• users access list as a series of 'passworduser' and 'rsauser entries.

Archives may include "mandatory" users that cannot be removed. They are typically used to add an enterprise wide recovery RSA key to all archives. Extreme care must be taken to protect these key, as it can decrypt all past archives generated from within that company. _catalog: file list This subpart is stored under index 4 ("_catalog") of the MS-OLEPS metadata. It contains a series of 'fileprops' TLV structures, one for each file or directory. The file hierarchy can be reconstructed by checking the 'parent_id' field of each file entry. If 'parent_id' is 0 then the file is located at the top-level of the hierarchy, otherwise it's located under the directory with the matching 'file_id'. TLV format This format is a series of fields :

• 4 bytes for Type (specified as a 4-bytes hexadecimal below)
• 4 bytes for value Length (uint32be)
• Value

Value semantics depend on its Type. It may contain an uint32be integer, a UTF-16LE string, a character sequence, or an inner TLV structure. Unless otherwise noted, TLV structures appear once. Some fields are optional and may not be present at all (e.g. 'archive_createdwith'). Some fields are unique within a structure (e.g. 'files_iv'), other may be repeated within a structure to form a list (e.g. 'fileprops' and 'passworduser'). The following top-level types that have been identified, and detailed in the next sections:

• 80110600: fileprops, used for the file list as well as for the global archive properties
• 001b0600: archive_extraprops
• 80140600: accesslist

Some additional unidentified types may be present. _ctlfile TLV

• 80110600: fileprops (TLV structure): global archive properties
  • 00230400: archive_pathname (UTF-16LE string): initial archive filename (past versions also leaked the full pathname of the initial archive)
  • 80270200: encryption_mode (utf32be): 103 for "AES-CBC-STREAM", 104 for "AES-CBC-CTS"
  • 80260200: encryption_strength (utf32be): AES key size, in bytes (e.g. 32 means AES with a 256-bit key)
  • 80280500: files_iv (sequence of bytes): global IV for all filenames and file contents
• 001b0600: archive_extraprops (TLV structure): additionnal archive properties (optional)
  • 00c40500: archive_creationtime (FILETIME): date and time when archive was initially created (optional)
  • 00c00400: archive_createdwith (UTF-16LE string): uuid-like structure describing the application that initialized the archive (optional)
    00000188-1000-3CA8-8868-36F59DEFD14D is Zed! Free 1.0.188.
• 80140600: accesslist (TLV structure): describe the users, their key encryption and their permissions
  • 80610600: passworduser (TLV structure): user identified by password (0 or more)
  • 80620600: rsauser (TLV structure): user identified by RSA key (via file or PKCS#11 token) (0 or more)
  • Fields common to passworduser and rsauser:
    • 80710400: login (UTF-16LE string): user name
    • 80720300: login_md5 (sequence of bytes): used by the application to search for a user name
    • 807e0100: priv1 (uchar): user privileges; present and set to 1 when user is admin (optional)
    • 00830200: priv2 (uint32be): user privileges; present and set to 2 when user is admin, present and set to 5 when user is a marked as mandatory, e.g. for recovery keys (optional)
    • 80740500: files_key_ciphertext (sequence of bytes): the archive encryption key, itself encrypted
    • 00840500: user_creationtime (FILETIME): date and time when the user was added to the archive
  • passworduser-specific fields:
    • 80760500: pbe_salt (sequence of bytes): salt for PBE
    • 80770200: pbe_iter (uint32be): number of iterations for PBE
    • 80780200: pkcs12_hashfunc (uint32be): hash function used for PBE and PBA key derivation
    • 80790500: pba_checksum (sequence of bytes): password derived with PBA to check for password validity
    • 807a0500: pba_salt (sequence of bytes): salt for PBA
    • 807b0200: pba_iter (uint32be): number of iterations for PBA
  • rsauser-specific fields:
    • 807d0500: certificate (sequence of bytes): user X509 certificate in DER format

_catalog TLV

• 80110600: fileprops (TLV structure): describe the archive files (0 or more)
  • 80300500: file_id (sequence of bytes): a 16-byte unique identifier
  • 80310400: filename_halfanon (UTF-16LE string): half-anonymized filename, e.g. File1.txt (leaking filename extension)
  • 00380500: filename_ciphertext (sequence of bytes): encrypted filename; may have a trailing NUL byte once decrypted
  • 80330500: file_size (uint64le): decompressed file size in bytes
  • 80340500: file_creationtime (FILETIME): file creation date and time
  • 80350500: file_lastwritetime (FILETIME): file last modification date and time
  • 80360500: file_lastaccesstime (FILETIME): file last access date and time
  • 00370500: parent_directory_id (sequence of bytes): file_id of the parent directory, 0 is top-level
  • 80320100: is_dir (uint32be): 1 if entry is directory (optional)

Decrypting the archive AES key rsauser The user accessing the archive will be authenticated by comparing his/her X509 certificate with the one stored in the 'certificate' field using DER format. The 'files_key_ciphertext' field is then decrypted using the PKCS#1 v1.5 encryption mechanism, with the private key that matches the user certificate. passworduser An intermediary user key, a user IV and an integrity checksum will be derived from the user password, using the deprecated PKCS#12 method as described at rfc7292 appendix B. Note: this is not PKCS#5 (nor PBKDF1/PBKDF2), this is an incompatible method from PKCS#12 that notably does not use HMAC. The 'pkcs12_hashfunc' field defines the underlying hash function. The following values have been identified:

• 21: SHA-1
• 22: SHA-256

PBA - Password-based authentication The user accessing the archive will be authenticated by deriving an 8-byte sequence from his/her password. The parameters for the derivation function are:

• ID: 3
• 'pba_salt': the salt, typically an 8-byte random sequence
• 'pba_iter': the iteration count, typically 200000

The derivation is checked against 'pba_checksum'. PBE - Password-based encryption Once the user is identified, 2 new values are derived from the password with different parameters to produce the IV and the key decryption key, with the same hash function:

• 'pbe_salt': the salt, typically an 8-bytes random sequence
• 'pbe_iter': the iteration count, typically 100000

The parameters specific to user key are:

• ID: 1
• size: 32

The user key needs to be truncated to a length of 'encryption_strength', as specified in bytes in the archive properties. The parameters specific to user IV are:

• ID: 2
• size: 16

Once the key decryption key and the IV are derived, 'files_key_ciphertext' is decrypted using AES CBC, with PKCS#7 padding. Identifying file streams The name of the MS-CFB stream is derived by shuffling the bytes from the 'file_id' field and then encoding the result as hexadecimal. The reordering is:

Initial  offset: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Shuffled offset: 3 2 1 0 5 4 7 6 8 9 10 11 12 13 14 15

The 16th byte is usually a NUL byte, hence the stream identifier is a 30-character-long string. Decrypting files The compressed stream is split in chunks of 512 bytes, each of them encrypted separately using AES CBS and the global archive encryption scheme. Decryption uses the global AES key (retrieved using the user credentials), and the global IV (retrieved from the deobfuscated archive metadata). The IV for each chunk is computed by:

• expressing the current chunk number as little endian on 16 bytes
• XORing it with the global IV
• encrypting with the global AES key in ECB mode (without IV).

Each chunk is an independent stream and the decryption process involves end-of-stream handling even if this is not the end of the actual file. This is particularly important for the CTS handler. Note: this is not to be confused with CTR block cipher mode of operation with operates differently and requires a nonce. Decompressing files Compressed streams are zlib stream with default compression options and can be decompressed following the zlib format. Test cases Excluded for brevity, cf. https://www.beuc.net/zed/#test-cases. Conventions and references

• uint32le: unsigned int on 4 bytes, little endian.
• uint32be: unsigned int on 4 bytes, big endian.
• uint64le: unsigned int on 8 bytes, little endian.
• uchar: unsigned char on 1 byte.
• (hexadecimal): represents 1 byte with 2 hexadecimal numbers.
  e.g. 4142 (hexadecimal) means decimal values 65 and then 66.
• (ascii): represents 1 byte with the matching 1-byte ASCII value.
  e.g. 'AB\0' means decimal values 65 then 66 and then 0.
• UTF-16LE: text encoded using 2-bytes Unicode, little-endian, without Byte Order Mark (BOM), as commonly used under MS Windows.
• IV: initialization vector for AES encryption and decryption.
• CBC: Cipher Block Chaining, see Recommendation for Block Cipher Modes of Operation and Wikipedia.
• CTS: CipherText Stealing, one type of end-of-stream handling, see Recommendation for Block Cipher Modes of Operation: Three Variants of Ciphertext Stealing for CBC Mode and Wikipedia.
• PBA, PBE: password-based authentication, password-based encryption; see PKCS #11 Cryptographic Token Interface Current Mechanisms Specification.
• MS-CFB: Compound File Binary Format, documented at MSDN, libolecf, Wikipedia.
  Libraries that can manipulate this format include olefile, libolecf and libforensics.
• MS-OLEPS: OLE Property Set, documented at MSDN and libfole.
  The 'oleps.py' demo from libforensics displays detailed information on MS-OLEPS streams; olefile and libolecf have basic support.
• TLV : Type-Length-Value encoding ; 4 bytes for type, 4 bytes for length (uint32be), and a variable-sized value; see Type-length-value at Wikipedia for the general principle.
• FILETIME: timestamp as 1/10000000s since January 1, 1601 (UTC), see FILETIME structure at MSDN.

Feedback Feel free to send comments at beuc@beuc.net. If you have .zed files that you think are not covered by this document, please send them as well (replace sensitive files with other ones). The author's GPG key can be found at 8FF1CB6E8D89059F. Copyright (C) 2017 Sylvain Beucler Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without any warranty.

TL,DR: I reverse-engineered the .zed encrypted archive format.
Following a clean-room design, I'm providing a description that can be implemented by a third-party.
Interested? (reference version at: https://www.beuc.net/zed/) .zed archive file format Introduction Archives with the .zed extension are conceptually similar to an encrypted .zip file. In addition to a specific format, .zed files support multiple users: files are encrypted using the archive master key, which itself is encrypted for each user and/or authentication method (password, RSA key through certificate or PKCS#11 token). Metadata such as filenames is partially encrypted. .zed archives are used as stand-alone or attached to e-mails with the help of a MS Outlook plugin. A variant, which is not covered here, can encrypt/decrypt MS Windows folders on the fly like ecryptfs. In the spirit of academic and independent research this document provides a description of the file format and encryption algorithms for this encrypted file archive. See the conventions section for conventions and acronyms used in this document. Structure overview The .zed file format is composed of several layers.

• The main container is using the (MS-CFB), which is notably used by MS Office 97-2003 .doc files. It contains several streams:
  • Metadata stream: in OLE Property Set format (MS-OLEPS), contains 2 blobs in a specific Type-Length-Value (TLV) format:
    • _ctlfile: global archive properties and access list
      It is obfuscated by means of static-key AES encryption.
      The properties include archive initial filename and a global IV.
      A global encryption key is itself encrypted in each user entry.
    • _catalog: file list
      Contains each file metadata indexed with a 15-bytes identifier.
      Directories are supported.
      Full filename is encrypted using AES.
      File extension is (redundantly) stored in clear, and so are file metadata such as modification time.
  • Each file in the archive compressed with zlib and encrypted with the standard AES algorithm, in a separate stream.
    Several encryption schemes and key sizes are supported.
    The file stream is split in chunks of 512 bytes, individually encrypted.
  • Optional streams, contain additional metadata as well as pictures to display in the application background ("watermarks"). They are not discussed here.

Or as a diagram:

+----------------------------------------------------------------------------------------------------+
  .zed archive (MS-CBF)                                                                               
                                                                                                      
   stream #1                         stream #2                       stream #3...                     
  +------------------------------+  +---------------------------+  +---------------------------+      
    metadata (MS-OLEPS)               encryption (AES)               encryption (AES)                 
                                      512-bytes chunks               512-bytes chunks                 
    +--------------------------+                                                                      
      obfuscation (static key)        +-----------------------+      +-----------------------+        
      +----------------------+       -  compression (zlib)     -    -  compression (zlib)     -       
       _ctlfile (TLV)                                                                            ...  
      +----------------------+          +---------------+              +---------------+               
    +--------------------------+          file contents                  file contents                
                                                                                                      
    +--------------------------+     -  +---------------+      -    -  +---------------+      -       
      _catalog (TLV)                                                                                  
    +--------------------------+      +-----------------------+      +-----------------------+        
  +------------------------------+  +---------------------------+  +---------------------------+      
+----------------------------------------------------------------------------------------------------+

Encryption schemes Several AES key sizes are supported, such as 128 and 256 bits. The Cipher Block Chaining (CBC) block cipher mode of operation is used to decrypt multiple AES 16-byte blocks, which means an initialisation vector (IV) is stored in clear along with the ciphertext. All filenames and file contents are encrypted using the same encryption mode, key and IV (e.g. if you remove and re-add a file in the archive, the resulting stream will be identical). No cleartext padding is used during encryption; instead, several end-of-stream handlers are available, so the ciphertext has exactly the size of the cleartext (e.g. the size of the compressed file). The following variants were identified in the 'encryption_mode' field. STREAM This is the end-of-stream handler for:

• obfuscated metadata encrypted with static AES key
• filenames and files in archives with 'encryption_mode' set to "AES-CBC-STREAM"
• any AES ciphertext of size < 16 bytes, regardless of encryption mode

This end-of-stream handler is apparently specific to the .zed format, and applied when the cleartext's does not end on a 16-byte boundary ; in this case special processing is performed on the last partial 16-byte block. The encryption and decryption phases are identical: let's assume the last partial block of cleartext (for encryption) or ciphertext (for decryption) was appended after all the complete 16-byte blocks of ciphertext:

• the second-to-last block of the ciphertext is encrypted in AES-ECB mode (i.e. block cipher encryption only, without XORing with the IV)
• then XOR-ed with the last partial block (hence truncated to the length of the partial block)

In either case, if the full ciphertext is less then one AES block (< 16 bytes), then the IV is used instead of the second-to-last block. CTS CTS or CipherText Stealing is the end-of-stream handler for:

• filenames and files in archives with 'encryption_mode' set to "AES-CBC-CTS".
  • exception: if the size of the ciphertext is < 16 bytes, then "STREAM" is used instead.

It matches the CBC-CS3 variant as described in Recommendation for Block Cipher Modes of Operation: Three Variants of Ciphertext Stealing for CBC Mode. Empty cleartext Since empty filenames or metadata are invalid, and since all files are compressed (resulting in a minimum 8-byte zlib cleartext), no empty cleartext was encrypted in the archive. metadata stream It is named 05356861616161716149656b7a6565636e576a33317a7868304e63 (hexadecimal), i.e. the character with code 5 followed by '5haaaaqaIekzeecnWj31zxh0Nc' (ASCII). The format used is OLE Property Set (MS-OLEPS). It introduces 2 property names "_ctlfile" (index 3) and "_catalog" (index 4), and 2 instances of said properties each containing an application-specific VT_BLOB (type 0x0041). _ctlfile: obfuscated global properties and access list This subpart is stored under index 3 ("_ctlfile") of the MS-OLEPS metadata. It consists of:

• static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by 0100 (hexadecimal) (18 bytes total)
• 16-byte IV
• ciphertext
• 1 uint32be representing the length of all the above
• static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by "ZoneCentral (R)" (ASCII) and a NUL byte (32 bytes total)

The ciphertext is encrypted with AES-CBC "STREAM" mode using 128-bit static key 37F13CF81C780AF26B6A52654F794AEF (hexadecimal) and the prepended IV so as to obfuscate the access list. The ciphertext is continuous and not split in chunks (unlike files), even when it is larger than 512 bytes. The decrypted text contain properties in a TLV format as described in _ctlfile TLV:

• global archive properties as a 'fileprops' structure,
• extra archive properties as a 'archive_extraprops' structure
• users access list as a series of 'passworduser' and 'rsauser entries.

Archives may include "mandatory" users that cannot be removed. They are typically used to add an enterprise wide recovery RSA key to all archives. Extreme care must be taken to protect these key, as it can decrypt all past archives generated from within that company. _catalog: file list This subpart is stored under index 4 ("_catalog") of the MS-OLEPS metadata. It contains a series of 'fileprops' TLV structures, one for each file or directory. The file hierarchy can be reconstructed by checking the 'parent_id' field of each file entry. If 'parent_id' is 0 then the file is located at the top-level of the hierarchy, otherwise it's located under the directory with the matching 'file_id'. TLV format This format is a series of fields :

• 4 bytes for Type (specified as a 4-bytes hexadecimal below)
• 4 bytes for value Length (uint32be)
• Value

Value semantics depend on its Type. It may contain an uint32be integer, a UTF-16LE string, a character sequence, or an inner TLV structure. Unless otherwise noted, TLV structures appear once. Some fields are optional and may not be present at all (e.g. 'archive_createdwith'). Some fields are unique within a structure (e.g. 'files_iv'), other may be repeated within a structure to form a list (e.g. 'fileprops' and 'passworduser'). The following top-level types that have been identified, and detailed in the next sections:

• 80110600: fileprops, used for the file list as well as for the global archive properties
• 001b0600: archive_extraprops
• 80140600: accesslist

Some additional unidentified types may be present. _ctlfile TLV

• 80110600: fileprops (TLV structure): global archive properties
  • 00230400: archive_pathname (UTF-16LE string): initial archive filename (past versions also leaked the full pathname of the initial archive)
  • 80270200: encryption_mode (utf32be): 103 for "AES-CBC-STREAM", 104 for "AES-CBC-CTS"
  • 80260200: encryption_strength (utf32be): AES key size, in bytes (e.g. 32 means AES with a 256-bit key)
  • 80280500: files_iv (sequence of bytes): global IV for all filenames and file contents
• 001b0600: archive_extraprops (TLV structure): additionnal archive properties (optional)
  • 00c40500: archive_creationtime (FILETIME): date and time when archive was initially created (optional)
  • 00c00400: archive_createdwith (UTF-16LE string): uuid-like structure describing the application that initialized the archive (optional)
    00000188-1000-3CA8-8868-36F59DEFD14D is Zed! Free 1.0.188.
• 80140600: accesslist (TLV structure): describe the users, their key encryption and their permissions
  • 80610600: passworduser (TLV structure): user identified by password (0 or more)
  • 80620600: rsauser (TLV structure): user identified by RSA key (via file or PKCS#11 token) (0 or more)
  • Fields common to passworduser and rsauser:
    • 80710400: login (UTF-16LE string): user name
    • 80720300: login_md5 (sequence of bytes): used by the application to search for a user name
    • 807e0100: priv1 (uchar): user privileges; present and set to 1 when user is admin (optional)
    • 00830200: priv2 (uint32be): user privileges; present and set to 2 when user is admin, present and set to 5 when user is a marked as mandatory, e.g. for recovery keys (optional)
    • 80740500: files_key_ciphertext (sequence of bytes): the archive encryption key, itself encrypted
    • 00840500: user_creationtime (FILETIME): date and time when the user was added to the archive
  • passworduser-specific fields:
    • 80760500: pbe_salt (sequence of bytes): salt for PBE
    • 80770200: pbe_iter (uint32be): number of iterations for PBE
    • 80780200: pkcs12_hashfunc (uint32be): hash function used for PBE and PBA key derivation
    • 80790500: pba_checksum (sequence of bytes): password derived with PBA to check for password validity
    • 807a0500: pba_salt (sequence of bytes): salt for PBA
    • 807b0200: pba_iter (uint32be): number of iterations for PBA
  • rsauser-specific fields:
    • 807d0500: certificate (sequence of bytes): user X509 certificate in DER format

_catalog TLV

• 80110600: fileprops (TLV structure): describe the archive files (0 or more)
  • 80300500: file_id (sequence of bytes): a 16-byte unique identifier
  • 80310400: filename_halfanon (UTF-16LE string): half-anonymized filename, e.g. File1.txt (leaking filename extension)
  • 00380500: filename_ciphertext (sequence of bytes): encrypted filename; may have a trailing NUL byte once decrypted
  • 80330500: file_size (uint64le): decompressed file size in bytes
  • 80340500: file_creationtime (FILETIME): file creation date and time
  • 80350500: file_lastwritetime (FILETIME): file last modification date and time
  • 80360500: file_lastaccesstime (FILETIME): file last access date and time
  • 00370500: parent_directory_id (sequence of bytes): file_id of the parent directory, 0 is top-level
  • 80320100: is_dir (uint32be): 1 if entry is directory (optional)

Decrypting the archive AES key rsauser The user accessing the archive will be authenticated by comparing his/her X509 certificate with the one stored in the 'certificate' field using DER format. The 'files_key_ciphertext' field is then decrypted using the PKCS#1 v1.5 encryption mechanism, with the private key that matches the user certificate. passworduser An intermediary user key, a user IV and an integrity checksum will be derived from the user password, using the deprecated PKCS#12 method as described at rfc7292 appendix B. Note: this is not PKCS#5 (nor PBKDF1/PBKDF2), this is an incompatible method from PKCS#12 that notably does not use HMAC. The 'pkcs12_hashfunc' field defines the underlying hash function. The following values have been identified:

• 21: SHA-1
• 22: SHA-256

PBA - Password-based authentication The user accessing the archive will be authenticated by deriving an 8-byte sequence from his/her password. The parameters for the derivation function are:

• ID: 3
• 'pba_salt': the salt, typically an 8-byte random sequence
• 'pba_iter': the iteration count, typically 200000

The derivation is checked against 'pba_checksum'. PBE - Password-based encryption Once the user is identified, 2 new values are derived from the password with different parameters to produce the IV and the key decryption key, with the same hash function:

• 'pbe_salt': the salt, typically an 8-bytes random sequence
• 'pbe_iter': the iteration count, typically 100000

The parameters specific to user key are:

• ID: 1
• size: 32

The user key needs to be truncated to a length of 'encryption_strength', as specified in bytes in the archive properties. The parameters specific to user IV are:

• ID: 2
• size: 16

Once the key decryption key and the IV are derived, 'files_key_ciphertext' is decrypted using AES CBC, with PKCS#7 padding. Identifying file streams The name of the MS-CFB stream is derived by shuffling the bytes from the 'file_id' field and then encoding the result as hexadecimal. The reordering is:

Initial  offset: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Shuffled offset: 3 2 1 0 5 4 7 6 8 9 10 11 12 13 14 15

The 16th byte is usually a NUL byte, hence the stream identifier is a 30-character-long string. Decrypting files The compressed stream is split in chunks of 512 bytes, each of them encrypted separately using AES CBS and the global archive encryption scheme. Decryption uses the global AES key (retrieved using the user credentials), and the global IV (retrieved from the deobfuscated archive metadata). The IV for each chunk is computed by:

• expressing the current chunk number as little endian on 16 bytes
• XORing it with the global IV
• encrypting with the global AES key in ECB mode (without IV).

Each chunk is an independent stream and the decryption process involves end-of-stream handling even if this is not the end of the actual file. This is particularly important for the CTS handler. Note: this is not to be confused with CTR block cipher mode of operation with operates differently and requires a nonce. Decompressing files Compressed streams are zlib stream with default compression options and can be decompressed following the zlib format. Test cases Excluded for brevity, cf. https://www.beuc.net/zed/#test-cases. Conventions and references

• uint32le: unsigned int on 4 bytes, little endian.
• uint32be: unsigned int on 4 bytes, big endian.
• uint64le: unsigned int on 8 bytes, little endian.
• uchar: unsigned char on 1 byte.
• (hexadecimal): represents 1 byte with 2 hexadecimal numbers.
  e.g. 4142 (hexadecimal) means decimal values 65 and then 66.
• (ascii): represents 1 byte with the matching 1-byte ASCII value.
  e.g. 'AB\0' means decimal values 65 then 66 and then 0.
• UTF-16LE: text encoded using 2-bytes Unicode, little-endian, without Byte Order Mark (BOM), as commonly used under MS Windows.
• IV: initialization vector for AES encryption and decryption.
• CBC: Cipher Block Chaining, see Recommendation for Block Cipher Modes of Operation and Wikipedia.
• CTS: CipherText Stealing, one type of end-of-stream handling, see Recommendation for Block Cipher Modes of Operation: Three Variants of Ciphertext Stealing for CBC Mode and Wikipedia.
• PBA, PBE: password-based authentication, password-based encryption; see PKCS #11 Cryptographic Token Interface Current Mechanisms Specification.
• MS-CFB: Compound File Binary Format, documented at MSDN, libolecf, Wikipedia.
  Libraries that can manipulate this format include olefile, libolecf and libforensics.
• MS-OLEPS: OLE Property Set, documented at MSDN and libfole.
  The 'oleps.py' demo from libforensics displays detailed information on MS-OLEPS streams; olefile and libolecf have basic support.
• TLV : Type-Length-Value encoding ; 4 bytes for type, 4 bytes for length (uint32be), and a variable-sized value; see Type-length-value at Wikipedia for the general principle.
• FILETIME: timestamp as 1/10000000s since January 1, 1601 (UTC), see FILETIME structure at MSDN.

Feedback Feel free to send comments at beuc@beuc.net. If you have .zed files that you think are not covered by this document, please send them as well (replace sensitive files with other ones). The author's GPG key can be found at 8FF1CB6E8D89059F. Copyright (C) 2017 Sylvain Beucler Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without any warranty.

Here is my monthly update covering what I have been doing in the free software world in August 2017 (previous month):

• Created ZeroCoolOS, a live operating system that plays the film Hackers (1995) on a continuous loop.
• Sent a patch for pristine-tar to allow storage of detached upstream signatures. (#871809)
• Worked more on Lintian, a static analysis tool for Debian packages, reporting on various errors, omissions and quality-assurance issues to the maintainer (previous changes):
  • Fix an apache2-unparsable-dependency false positive by allowing periods in dependency names. (#873701)
  • Ignore "repacked" packages when checking for upstream source tarball signatures as they will never match.
  • Downgrade the severity of orig-tarball-missing-upstream-signature. (#870722)
  • From a suggestion by Theodore Ts'o, expand the explanation of orig-tarball-missing-upstream-signature to include the location of where dpkg-source looks.
  • Address a number of issues in the copyright-year-in-future tag including preventing false positives in port numbers, email addresses, ISO standard numbers and street addresses (#869788), as well as "meta" or testing statements (#873323). In addition, report all violating years in a line and expand the testsuite.
  • Don't match quoted "FIXME" variants of file-contains-fixme-placeholder (#870199), avoid checking copyright_hints files (#872843) and downgrade the tag's severity.
  • Apply a patch from Alex Muntada to recommend "substr" over of "substring" in mentions-deprecated-usr-lib-perl5-directory. (#871767)
  • Prevent missing-build-dependency-for-dh_-command false positives exposed by following the advice in useless-autoreconf-build-depends. (#869541)
  • Ensure readme-debian-contains-debmake-template also checks for files containing "Automatically generated by debmake".
  • Check python3-foo packages have a Section: python, not just python2-foo. (#870272)
  • Check for packages shipping compiled Java class files. (#873211)
  • Additionally consider .cljc files to avoid codeless-jar warnings. (#870649)
  • Prevent desktop-entry-lacks-keywords-entry false positives for Link and Directory-style .desktop files. (#873702)
  • Split out Python checks from checks/scripts.pm check to a new, source check of type source.
  • Check for python-foo without a corresponding python3-foo package. (#870681)
  • Complain about packages that Build-Depend on python-sphinx only. (#870730)
  • Warn about packages that alternatively Build-Depend on the Python 2 and Python 3 versions of Sphinx. (#870758)
  • Check for packages that depend on Python 2.x. (#870822)
  • Correct false positives in unconditional-use-of-dpkg-statoverride by detecting "if !" as a shell prefix. (#869587)
  • Alert on for missing calls to dpkg-maintscript-helper(1) in maintainer scripts. (#872042)
  • Check for packages using sensible-utils without declaring a dependency after splitting from debianutils. (#872611)
  • Warn about scripts using nodejs as an interpreter now that the nodejs script provides /usr/bin/node. (#873096)
  • Remove recommendations to add a Testsuite: autopkgtest field to debian/control and emit a new tag the package if it does so. (#865531)
  • Recognise autopkgtest-pkg-elpa as a valid test suite. (#873458)
  • Add note to /etc/bash_completion.d's obsolete path warning output regarding stricter filename requirements. (#814599)
  • Add 4.0.1 and 4.1.0 as known Policy standards versions.
  • Apply a patch from Maia Everett to avoid British spellings under the en_US locale. (#868897)
  • Stop emitting maintainer,uploader -address-causes-mail-loops for @packages.debian.org addresses. (#871575)
  • Modify Lintian::Data's all subroutine to always return keys in insertion order.
  • Apply a patch from Steve Langasek to accomodate binutils outputting symbols in a different format on the ppc64el architecture. (#869750)
  • Add an explicit test for packages including external fonts via the Google Font and TypeKit APIs. (#873434)
  • Add missing entries in internal Test-For fields to make development/testing workflow less error-prone.
• Sent three pull requests to git-buildpackage, a tool to assist in Debian packaging from Git repositories:
  • Make pq --abbrev= configurable. (#872351)
  • Use build profiles to avoid installation of test dependencies. (#31)
  • Correct "allow to" grammar. (#30)
• Updated travis.debian.net (my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform for testing):
  • Move away from deb.debian.org; Travis appears to be using a HTTP proxy that strips SRV records. (commit)
  • Highlight double quotes are required for TRAVIS_DEBIAN_EXTRA_REPOSITORY. (commit)
  • Use force-unsafe-io. (commit)
  • Clarify docs when upstream already has a travis.yml file. (#46)
  • Make documentation easier to copy-paste. (commit)
• Merged a pull request in django-slack, my library to easily post messages to the Slack group-messaging utility, where instantiation of a SlackException was failing. (#71)
• Assigned two pull requests to the Redis key-value database store to correct "did not received" and "faield" typos. (#4216 & #4215).

---

Today I noticed the internet buzz about a new alleged Linux malware called OutlawCountry by the CIA, and leaked by Wikileaks. The malware redirects traffic from the victim to a control server in order to spy or whatever. To redirect this traffic, they use simple Netfilter NAT rules injected in the kernel. According to many sites commenting on the issue, is seems that there is something wrong with the Linux kernel Netfilter subsystem, but I read the leaked docs, and what they do is to load a custom kernel module in order to be able to load Netfilter NAT table/rules with more priority than the default ones (overriding any config the system may have). Isn t that clear? The attacker is loading a custom kernel module as root in your machine. They don t use Netfilter to break into your system. The problem is not Netfilter, the problem is your whole machine being under their control. With root control of the machine, they could simply use any mechanism, like kpatch or whatever, to replace your whole running kernel with a new one, with full access to memory, networking, file system et al. They probably use a rootkit or the like to take over the system.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git. Get it on github. These release notes are also available on the git-cinnabar wiki. What s new since 0.4.0?

• git-cinnabar-helper is now mandatory. You can either download one with git cinnabar download on supported platforms or build one with make helper.
• Metadata changes require to run git cinnabar fsck.
• Mercurial tags are consolidated in a separate (fake) repository. See the README file.
• Updated git to 2.13.0 for git-cinnabar-helper.
• Improved memory consumption and performance.
• Improved experimental support for pushing merges.
• Experimental support for clonebundles.
• Removed support for the .git/hgrc file for mercurial specific configuration.
• Support any version of Git (was previously limited to 1.8.5 minimum)